Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release-21.2: sqlliveness: session expiry callbacks must be async #71433

Merged
merged 1 commit into from
Oct 12, 2021
Merged

release-21.2: sqlliveness: session expiry callbacks must be async #71433

merged 1 commit into from
Oct 12, 2021

Conversation

dhartunian
Copy link
Collaborator

Backport 1/1 commits from #71419.

/cc @cockroachdb/release

Previously, the sqlliveness session expiry callbacks were called in the
heartbeatLoop thread which executed the renew/expiry logic. This could
cause deadlock since session expiration is used to trigger a shutdown of
the SQL instance via stopper.Stop(). The stopper would wait for all
async tasks to quiesce, but the heartbeatLoop would continue, waiting
for the callbacks to finish running. In addition, this task would hold a
lock on l.mu while waiting for the callbacks to run causing other
threads to wait if they needed to retrieve the session.

This change invokes each callback in its own goroutine to prevent this
deadlock.

Resolves #71292

Release note: None

@dhartunian
sqlliveness: session expiry callbacks must be async
b503ade 
Previously, the sqlliveness session expiry callbacks were called in the
heartbeatLoop thread which executed the renew/expiry logic. This could
cause deadlock since session expiration is used to trigger a shutdown of
the SQL instance via `stopper.Stop()`. The stopper would wait for all
async tasks to quiesce, but the `heartbeatLoop` would continue, waiting
for the callbacks to finish running. In addition, this task would hold a
lock on `l.mu` while waiting for the callbacks to run causing other
threads to wait if they needed to retrieve the session.

This change invokes each callback in its own goroutine to prevent this
deadlock.

Resolves cockroachdb#71292

Release note: None
@dhartunian dhartunian requested review from ajwerner, jaylim-crl and a team October 11, 2021 23:54
@blathers-crl
Copy link

blathers-crl bot commented Oct 11, 2021 •
edited by dhartunian
Loading

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Patches should only be created for serious issues.
  • Patches should not break backwards-compatibility.
  • Patches should change as little code as possible.
  • Patches should not change on-disk formats or node communication protocols.
  • Patches should not add new functionality.
  • Patches must not add, edit, or otherwise modify cluster versions; or add version gates.
If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

  • What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
  • Will this work in a cluster of mixed patch versions? Did we test that?
  • If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

@cockroach-teamcity
Copy link
Member

This change is Reviewable

ajwerner
ajwerner approved these changes Oct 12, 2021
View reviewed changes
Copy link
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @jaylim-crl)

@dhartunian dhartunian merged commit 5377f3d into cockroachdb:release-21.2 Oct 12, 2021
@dhartunian dhartunian deleted the backport21.2-71419 branch October 12, 2021 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ajwerner ajwerner ajwerner approved these changes

@jaylim-crl jaylim-crl Awaiting requested review from jaylim-crl

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dhartunian @cockroach-teamcity @ajwerner